When Random Assignment Is Not Enough: Accounting for Item Selectivity in Experimental Research

A common goal in marketing research is to understand how one evaluates products that have been filtered through some type of screening or selection process. Typical examples include post-choice satisfaction ratings, certain free recall tasks, or the development of consideration sets followed by brand choice. In such situations, behavior is contingent not only on the alternatives being evaluated, but the choice context in which they have become available, creating differing degrees of selectivity. In this paper, we consider situations in which a polytomous selection process limits which items offer up subsequent information. We propose that not flexibly modeling the correlation between choice and evaluation across conditions can lead to systematic, erroneous interpretations of covariate effects on evaluation. We illustrate this by analyzing two experiments in which subjects choose among, and then evaluate, a frequently-purchased consumer good, as well as data first examined by Ratner, Kahn and Kahneman (1999). Results indicate not only strong selectivity effects, but that traditional specifications of the choice process – presuming the degree of selectivity is invariant across choice contexts – can lead to markedly different interpretations of variable effects. Our findings show that the size and composition of a choice set affects the degree of correlation between choice and evaluation, and further suggest that foregone alternatives can play a disproportionate role in selectivity. Moreover, failing to account for selectivity across experimental conditions, even in well-designed experimental settings, can lead to inaccurate substantive inferences about consumers’ choice processes.